Cluster key synchronization

ABSTRACT

Apparatus and method for synchronizing encryption keys among a cluster of security appliances and stand alone lifetime key management, LKM, appliances. The cluster includes security appliances where new encryption keys are generated and assigned to an SNS ID with an SNS CTR (counter). The security appliances inside a cluster have local sequence counters and share their keys. One security appliance is a coordinator with which the LKMs will synchronize. Each LKM also has a SNS ID and local sequence counter from which increasing sequence numbers are generated. In each security appliance in a cluster, the up-to-date stored sets of keys are organized with respect to SNS IDs and SNS CTRs associated with the other cluster members. The object keys are stored in the SNS space and a peer map associates a given peer with a given SNS ID, and version numbers are assigned and incremented when a key is modified.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is related to commonly owned U.S. patentapplications: patent application Ser. No. 11/740,474 entitled,“Apparatus for Lifetime Key Management,” by Hristo Bojinov et al., filedApr. 26, 2007; and U.S. patent application Ser. No. 11/740,490 entitled,“Peer to Peer Key Synchronization,” by Hristo Bojinov et al., filed Apr.26, 2007. These related applications are hereby incorporated herein byreference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to secure systems, and more specifically,to clustered security systems communicating with lifetime key management(LKM) appliances, wherein encryption keys are distributed andsynchronized among the processing entities in the system.

2. Background

A storage system is a computer that provides storage service relating tothe organization of information on writable persistent storage devices,such as memories, tapes or disks. The storage system is commonlydeployed within a storage area network (SAN) or a network attachedstorage (NAS) environment. When used within a NAS environment, thestorage system may be embodied as a file server including an operatingsystem that implements a file system to logically organize theinformation as a hierarchical structure of data containers, such asfiles on, e.g., the disks. Each “on-disk” file may be implemented as aset of data structures, e.g., disk blocks, configured to storeinformation, such as the actual data (i.e., file data) for the file.

The file server, or filer, may be further configured to operateaccording to a client/server model of information delivery to therebyallow many client systems (clients) to access shared resources, such asfiles, stored on the filer. Storage of information on a NAS system istypically deployed over a computer network comprising a geographicallydistributed collection of interconnected communication links, such asEthernet, that allow clients to remotely access the information (files)on the filer. The clients typically communicate with the filer byexchanging discrete frames or packets of data according to predefinedprotocols, such as the Transmission Control Protocol/Internet Protocol(TCP/IP).

In the client/server model, the client may comprise an applicationexecuting on a computer that “connects” to the filer over a computernetwork, such as a point-to-point link, shared local area network, widearea network or virtual private network implemented over a publicnetwork, such as the Internet. NAS systems generally utilize file-basedaccess protocols; therefore, each client may request the services of thefiler by issuing file system protocol messages (in the form of packets)to the file system over the network identifying one or more files to beaccessed without regard to specific locations, e.g., blocks, in whichthe data are stored on disk. By supporting a plurality of file systemprotocols, such as the conventional Common Internet File System (CIFS)and the Network File System (NFS) protocols, the utility of the filermay be enhanced for networking clients.

A SAN is a high-speed network that enables establishment of directconnections between a storage system and its storage devices. The SANmay thus be viewed as an extension to a storage bus and, as such, anoperating system of the storage system enables access to stored datausing block-based access protocols over the “extended bus”. In thiscontext, the extended bus is typically embodied as Fibre Channel (FC) orEthernet media adapted to operate with block access protocols, such asSmall Computer Systems Interface (SCSI) protocol encapsulation over FC(e.g., FCP) or TCP (iSCSI).

SCSI is a peripheral input/output (I/O) interface with a standard,device independent protocol that allows different peripheral devices,such as disks, to attach to a storage system. In SCSI terminology,clients operating in a SAN environment are “initiators” that initiatecommands and requests to access data. The storage system is thus a“target” configured to respond to the data access requests issued by theinitiators in accordance with a request/response protocol. Theinitiators and targets have endpoint addresses that, in accordance withthe FC protocol, comprise worldwide names (WWN). As known to thoseskilled in the art, a WWN is a unique identifier, e.g., a node name or aport name, consisting of an 8-byte number.

A SAN arrangement or deployment allows decoupling of storage from thestorage system, such as an application server, and some level ofinformation storage sharing at the storage system level. There are,however, environments wherein a SAN is dedicated to a single storagesystem. In some SAN deployments, the information is organized in theform of databases, while in others, a file-based organization isemployed. Where the information is organized as files, the clientrequesting the information maintains file mappings and manages filesemantics, while its requests (and storage system responses) address theinformation in terms of block addressing on disk using, e.g., a logicalunit number (lun).

A network environment may be provided, wherein information (data) isstored in secure storage served by one or more storage systems coupledto one or more security appliances. Each security appliance isconfigured to transform unencrypted data (cleartext) generated byclients (or initiators) into encrypted data (ciphertext) destined forsecure storage or “cryptainers” on the storage system (or target). Asused herein, a cryptainer is a piece of storage on a storage device,such as a disk, in which the encrypted data is stored. In the context ofa SAN environment, a cryptainer can be, e.g., a disk, a region on thedisk or several regions on one or more disks that, in the context of aSAN protocol, is accessible as a lun. In the context of a NASenvironment, the cryptainer may be a collection of files on one or moredisks. Specifically, in the context of the CIFS protocol, the cryptainermay be a share, while in the context of the NFS protocol, the cryptainermay be a mount point. In a tape environment, the cryptainer may be atape containing a plurality of tape blocks.

Each cryptainer is associated with its own encryption key, e.g., acryptainer key, which is used by the security appliance to encrypt anddecrypt the data stored on the cryptainer. An encryption key is a codeor number which, when taken together with an encryption algorithm,defines a unique transformation used to encrypt or decrypt data. Dataremains encrypted while stored in a cryptainer until requested by anauthorized client. At that time, the security appliance retrieves theencrypted data from the cryptainer, decrypts it and forwards theunencrypted data to the client.

Security systems often contain many cryptainers that are to be managedby a single security appliance or by multiple security appliancesarranged, for example in a cluster, employing and sharing encryptionkeys (herein “key” and “encryption key” are used interchangeably) witheach other and with non-clustered systems. When a cluster of multiplesecurity appliances is managed, often a LKM (Lifetime Key Management)appliance is a resource. However, some systems employ multiple LKMs and,correspondingly, the multiple LKMs are synchronized. As a result, anadministrator may simply not be able to coordinate the management andsynchronizations needed.

Another limitation of clustered security appliances occurs when keys(key objects) are created on one security appliance, but that appliancethen becomes inactive. In such a case, objects created may not besynchronized throughout the security appliance cluster and LKMs.

SUMMARY OF THE INVENTION

The limitations described above are addressed by a system and protocolfor synchronizing objects, illustratively key objects, among a clusterof security appliances and standalone LKMs (Lifetime Key Management), byassigning sequence numbers to key objects and storing those objects inidentified spaces. The sequence numbers allow synchronization of theidentified spaces, as compared to synchronizing appliances. Eachidentified space has a sequence counter associated with it, where thesequence counter tracks the number of key objects synchronized.Illustratively, only those key objects with sequence numbers equal to orhigher than the sequence number held in another appliance aretransferred during synchronization.

An identified space in the illustrative examples discussed herein is apersistent (sometimes referred to as non-volatile) group of memorylocations that are identified by a unique identification number (ID).The ID is used to distinguish the contents of other identified spaces.

An example of a cluster as used herein is a group of security appliancesthat elect one security appliance to act as a leader or coordinator. Thecluster is characterized by sharing the encryption keys generated withinany one security appliance among all the security appliances.Illustratively, stand alone LKM appliances synchronize only with thecoordinator security appliance.

Herein and as used below, when an appliance, such as an LKM communicateswith another LKM, the latter is defined herein as a “peer.” LKMs and/orsecurity appliances, illustratively, communicate over a link and embodythe only “nodes” in the system shown. In such an arrangement, each LKMat each node may have a series of identified spaces for storing keyobjects and associated counters that are synchronized with their“peers.”

The present invention is described below with respect to encryption keysin secure storage systems, but as known to those skilled in the art,wherever objects are distributed among a cluster of appliances andmanaged by an appliance, the present invention may be advantageouslyapplied. The scope of the present invention is not to be restricted tosecure storage systems alone. As used herein, the term “key object” mayinclude the key itself and other fields or attributes, e.g., a sequencenumber, a version number, a universally unique (for example an 16 byteor 124 bits may be used that defines very many numbers) identificationnumber, and other fields as may be applicable in other environments.Generally, “key” refers to the actual encryption key while “key object”refers to the key and the other above fields that may accompany the keywhen stored. However, “key” and “key object” are used interchangeably inthe art, but, in context, understood by those skilled in the art.

In a cluster of security appliance nodes, each node in the cluster has aunique identification and a separate identified space for storing keyobjects and a local sequence number space counter. Those key objects areassigned unique object identification and sequence numbers. Each keyobject, inserted into a node is identified by that node's ID, its ownunique object identification number and a sequence number generated fromits local sequence counter. In some applications other fields, e.g.,version numbers, etc. may be appended to the above data fields within akey object. That newly inserted object, with the node's ID, the sequencenumber and any other appended fields, is then sent to all the nodes inthe cluster. At the recipient node, each object is stored along with thesending node's (peer identification) ID, the received sequence number,the object's unique identification and any other fields. Since all thekey objects are identified and stored in all the nodes, any one node maybe used as a source for any key object. In this manner, the system isable to advantageously synchronize objects belonging to all theclustered nodes, even if a node goes offline.

In the above cluster comprising, for example, security appliances, asingle appliance stores all the objects in the system (including objectsof security appliances that have gone off line), and a standalone node,say an LKM, can receive key objects from that single security appliance.Illustratively, one on-line appliance may be elected as a leader orcoordinator of the cluster, where that coordinator is the only appliancethat will respond to a request for key objects from an LKM.

Illustratively, the organization of the different sequences within eachnode is identified by an SNS ID (Sequence Number Space Identifier) andan associated SNS CTR (counter). Each object within the SNS will have aunique Object Identifier. Each SNS ID is a persistent, globally uniqueidentifier.

In order to ensure that the LKM reboots, e.g., after a power failure,the SNS IDs and the SNS CTRs are stored persistently in the LKM. Ifthere is a power failure and reboot, the key synchronization resumeswhere it is left off instead of starting from the beginning.

According to an embodiment of the present invention, SNS IDs are beingsynchronized. Illustratively, the synchronization of nodes in theclustered environment, with SNS IDs, entails a process that may besplit, illustratively, into two processes. The first process, referredto as the “SNS Sync Process,” includes one node periodically queryinganother peer node and requesting the peer node's list of SNS IDs. Foreach node there is one SNS sync process. The peer node returns its listof SNS IDs. Since this list may contain the duplicate SNS IDs (returnedearlier from other peers) that are discarded.

Illustratively, when a new SNS ID is returned to the requesting node,the received SNS ID is stored and an SNS CTR is created and set to zeroAs described herein, objects may then be received from this peer nodeand stored in the identified space (SNS ID) of the receiver.

In order to better organize and facilitate the processes, a peer map maybe generated as a runtime structure that is maintained by the SNS SyncProcess, and used by the “Object Sync Process” to route requests to theproper peers. An entry in this map includes the SNS ID and the peer thatlast returned that SNS ID during the SNS Sync Process. When an SNS ID isreturned during the SNS Sync Process that is known to the requestingnode, the peer map is updated.

The Object Sync Process operates to synchronize objects in a given SNSID from a peer to a requesting node. Using the SNS ID as an entry intothe Peer Map, the associated peer ID is found. In practice, the ObjectSync Process synchronizes a stand alone node with another stand alonenode or with the coordinator security appliance mentioned above.

For example, consider that from the SNS Sync Process, node N1 receivesan SNS ID that N1 associates with peer P1 from the Peer Map. N1, sendsout the SNS ID and N1's SNS CTR contents, say C, to P1. As discussedbelow in more detail, the difference between the SNS CTR contents at P1and C represents the number of objects that are to be sent tosynchronize with N1.

The present invention provides a local sequence counter in each nodefrom which all sequence numbers for objects in SNS ID spaces areallocated. The counters are monotonically increasing counters, so thatno two objects share the same sequence number.

As a new object, typically an encryption key object, is added or anexisting encryption key object modified, the content of the localsequence counter is assigned to that object and then incremented,illustratively monotonically, in anticipation of receiving the nextobject. In addition a version number may be associated with the keyobject. When an existing encryption key object is modified, the versionnumber is incremented and the local sequence counter content assigned tothe encryption key object. The local sequence counter is thenincremented. The sequence counter is ever increasing.

The present invention is described below with respect to encryption keysin secure storage systems, but, wherever objects are distributed among acluster of systems and non-clustered nodes, the present invention may beadvantageously applied. The scope of the invention is not to berestricted to secure storage systems alone.

It will be appreciated by those skilled in the art that although thefollowing Detailed Description will proceed with reference being made toillustrative embodiments, the drawings, and methods of use, the presentinvention is not intended to be limited to these embodiments and methodsof use. Rather, the present invention is of broad scope and is intendedto be defined as only set forth in the accompanying claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention description below refers to the accompanying drawings, ofwhich:

FIG. 1 is a schematic block diagram system of a multi-protocol preferredembodiment employing the present invention;

FIG. 2 is a schematic block diagram of a cluster of multi-protocolsecurity appliances that may be advantageously used with the presentinvention;

FIG. 3 is a schematic block diagram of a lifetime key management (LKM)server in accordance with the present invention;

FIG. 4 is one possible data structure for an object;

FIG. 5 is an illustrative Peer Map;

FIG. 6 flow chart of an SNS ID synchronization process (SNS SyncProcess) in accordance with the present invention; and

FIG. 7 is a flow chart of an Object Sync Process in accordance with thepresent invention.

DETAILED DESCRIPTION OF AN ILLUSTRATIVE EMBODIMENT

A. Security Appliance Environment

FIG. 1 is a schematic block diagram of an environment 100 including acluster of multi-protocol security appliance 200 that may beadvantageously used with the present invention. Each security appliance200, 200′ is coupled between one or more clients 102 and one or morestorage systems 110. The security appliance 200, which is configured toact as an encryption proxy, intercepts a data access request issued byclient 102 and destined for the storage system 110, wherein the dataaccess request may be a read request to retrieve certain data stored onstorage devices, such as disks 120, coupled to the storage system 110 ora write request to store data on the disks. In the case of a writerequest, the security appliance 200 intercepts the request, encrypts thedata associated with the request and forwards the encrypted data to thestorage system 110 for storage at a specified location (address) on disk120. In the case of a read request, the security appliance interceptsthe request and forwards it onto the storage system, which returns therequested data to the appliance in encrypted form. The securityappliance 200 then decrypts the encrypted data and returns the decrypteddata to the client 102.

The storage system 110 is configured to provide storage service for bothfile and block protocol access to information stored on storage devices,such as disks 120. The storage system 110 is illustratively embodied asa system comprising a processor, a memory, a plurality of networkadapters, and a storage adapter (these components are not shown in FIG.1). The storage system 110 also includes a storage operating system (notshown) that includes a virtualization system including a file system tologically organize the information as a hierarchical structure of nameddirectories, file and virtual disk (vdisk) storage objects on disks 120.The storage system 110 includes the interface to communicate with thesecurity appliances 200, 200′ and the LKM's 300, 300′.

In the illustrative embodiment, the security appliance employs aconventional encryption algorithm, e.g., the Advanced EncryptionStandard (AES) or other appropriate algorithms, to transform unencrypteddata (cleartext) generated by the clients 102 into encrypted data(ciphertext) intended for secure storage, i.e., one or more cryptainers,on the storage system 110. To that end, the security applianceillustratively uses a high-quality, software or hardware-based pseudorandom number generation technique to generate encryption keys. Theencryption and decryption operations are performed using theseencryptions keys, such as a cryptainer key associated with eachcryptainer. As described herein, the security appliance 200 uses anappropriate cryptainer key to encrypt or decrypt portions of data storedin a particular cryptainer. In addition to performing encryption anddecryption operations, the security appliance 200 also performs accesscontrol, authentication, virtualization, and secure-logging operations.

A lifetime key management (LKM) appliance 300 is configured to manageall encryption keys used by the security appliances to encrypt anddecrypt data securely stored on the storage system 110, ensuringencryption key availability for the life of the secured data. Forexample, the LKM appliance 300 receives encrypted cryptainer keys fromthe security appliance 200 and sends encrypted cryptainer keys on demandto the appliance. The LKM appliance is further configured to support thecluster of security appliances 200, 200′, such that, when a particularappliance encounters a data access request directed to a cryptainer forwhich it does not have the appropriate key, that appliance accesses theLKM appliance 300 to receive the appropriate key.

In one embodiment, the LKM appliance 300 and the security appliance 200are in communication with other security appliances 200′ and with otherLKM appliances 300′. The other security appliances 200′, in turn,connect to other clients 102′. As noted earlier, the security appliancesand LKM appliances communicate over a link and embody “nodes.” Clients,storage systems, database servers, and like hardware and software shownherein are not nodes. Furthermore, when one node communicates withanother node, say via a secure link, that latter node is a “peer.” If,for example, the security appliance 200 communicates with a securityappliance 200′, the security appliance 200′ is a peer of the securityappliance 200. Similarly, if the LKM 300 communicates with an LKM 300′,LKM 300′ is a peer of LKM 300, and if an LKM 300 communicates with asecurity appliance 200, that security appliance 200 is a peer of the LKM300. When an LKM appliance communicates with a security appliance, thesecurity appliance sends its new or newly modified keys to the LKMappliance. In this embodiment, the security appliance requests andreceives up-to-date keys from an LKM as requested by a client.

The security appliances 200 and 200′ are included within a cluster. LKMs300 and 300′ are standalone nodes that are not part of any cluster.

A leader or coordinator security appliance is automatically elected bythe on-line (active) members of the cluster. The LKM appliances 300,300′ synchronizes key objects with the coordinator security appliance200, 200′ and thus obtains key objects from the entire cluster ofsecurity appliances. A trust relationship is established between eachsecurity appliance and LKM appliance using, e.g., a shared secret orcertificate to establish a secure communications channel. Once thesecure channel is established, any key generated by a security appliancecan be forwarded t via the coordinator to the LKM appliance 300, for,e.g., storage in a key database (not shown). Similarly, when an inquiryfrom a client requires a key not found in the security appliance, thatsecurity appliance obtains the key from the LKM. Keys are created and/ormodified in the various security appliances, and subsequentlytransferred to the LKMs. The LKMs synchronize the key objects tosuitably service the security appliances.

The LKM appliance 300 is interconnected to a database server 130, whichincludes additional storage devices such as disks 120. Alternatively, asecond storage system 110 may be interconnected with the LKM appliance300. In an alternative embodiment of the present invention, the LKMappliance 300 may utilize the external database server 130 for storageof keys. Thus, in such an embodiment, the database server 130 mayexecute a database application (not shown) implementing, e.g., aconventional SQ-L compliant database. Illustratively, the LKM appliance300 includes a set of internal storage 380 as shown in FIG. 3 describedfurther below. However, in alternative embodiments, the LKM appliance300 may utilize external storage, such as that offered by storage system110 of FIG. 1. It should be noted that while the LKM appliance 300 isshown interconnected to a database server 130 and a storage system 110,these are shown for illustrative purposes only. As such, neitherdatabase server 130 nor storage system 110 is necessary for operation ofa LKM appliance 300.

Furthermore, in embodiments where the LKM appliances 300, 300′ areimplemented within a single environment, each LKM appliance may utilizea different storage technique. Thus, for example, a first LKM appliance300 may utilize internal storage, a second LKM appliance 300′ mayutilize a database server 130 for key storage, while a third LKMappliance 300′ utilizes external storage provided by storage system 110.As can be appreciated by one skilled in the art, by the use of a LKMappliance 300, administrators are provided with additional configurationcapabilities to meet specific environmental and/or security concerns.

B. Security Appliance

FIG. 2 is a schematic block diagram of the multi-protocol securityappliance 200 that may be advantageously used with the presentinvention. As used herein, a security appliance denotes a computerhaving features such as simplicity of security service management forusers (system administrators) and clients of network attached storage(NAS) and storage area network (SAN) deployments. The security appliancecomprises one or more processors, e.g., central processing units (CPU202 a,b), a memory 210, one or more network adapters 220 a, b, a storageencryption processor (SEP) 290, and a card reader 230 interconnected bya system bus 240, such as a conventional Peripheral ComponentInterconnect (PCI) bus. The SEP 290 is configured to perform allencryption and decryption operations for the security appliance in asecure manner; for example, the SEP is configured to protect plaintextencryption keys from system software executing on each CPU 202.Accordingly, the SEP is illustratively embodied as a Federal InformationProcessing Standards (FIPS) certified module that is mounted onto adedicated interface card or other similar card.

Since the SEP 290 protects encryption keys from being “touched”(processed) by the system software executing on the CPU 202, a mechanismis needed to load keys into the SEP and retrieve keys from the SEP. Tothat end, the card reader 230 provides an interface between a “smart”system card 250 and the SEP 290 for purposes of exchanging encryptionkeys. Illustratively, the system card is a FIPS certified card that isconfigured with customized software code. Note that the SEP protectionprevents the operating system and/or other system software fromcompromising the security of the systems described herein.

The network adapters 220 couple the security appliance 200 between oneor more clients 102 and one or more storage systems 110 overpoint-to-point links, wide area networks, virtual private networksimplemented over a public network (Internet) or shared local areanetworks. In a SAN environment configured to support various SmallComputer Systems Interface (SCSI)-based data access protocols, includingSCSI encapsulated over TCP (iSCSI) and SCSI encapsulated over FC (FCP),the network adapters 220 may comprise host bus adapters (HBAs) havingthe mechanical, electrical and signaling circuitry needed to connect theappliance 200 to, e.g., a FC network. In a NAS environment configured tosupport, e.g., the conventional Common Internet File System (CIFS) andthe Network File System (NFS) data access protocols, the networkadapters 220 may comprise network interface cards (NICs) having themechanical, electrical and signaling circuitry needed to connect theappliance to, e.g., an Ethernet network.

The memory 210 illustratively comprises storage locations that areaddressable by the processors and adapters for storing software programsand data structures associated with the present invention. The processorand adapters may, in turn, comprise processing elements and/or logiccircuitry configured to execute the software programs and manipulate thedata structures. An operating system 212, portions of which is typicallyresident in memory and executed by the processing elements, functionallyorganizes the appliance 200 by, inter alia, invoking security operationsin support of software processes and/or modules implemented by theappliance. It will be apparent to those skilled in the art that otherprocessing and memory means, including various computer readable media,may be used for storing and executing program instructions pertaining tothe invention described herein.

Each clustered security appliance 200, 200′ can generate encryption keysusing a local key object generation process. This process, in turn,utilizes a local sequence counter 221, a local SNS (Sequence NumberSpace), and an SNS ID (an Identifier for that space) 213. This local keyobject generation serves to create encryption keys in response toclients' requests and store those keys in an identified space (SNS ID),together with a sequence counter representing the last key object socreated in that security appliance. In addition, key objects createdwithin other security appliances within a cluster are transferred andstored within each of the security appliances within the cluster. Forexample, SNS ID, SNS CTR and the objects created in, e.g., securityappliance A 215 are transferred into the memory 210 of securityappliance within the cluster. Similarly, key objects, generated in theclustered security appliance B and the other clustered securityappliances 217, are transferred and stored in each security appliance'smemory 210. As a result, each clustered security appliance has a list ortable including SNS IDs, SNS CTRs, and key objects for all the clusteredsecurity appliances, including its own objects. An Object ReplicationProcess 219 defines the process of transferring of key objects among thesecurity appliances. In one illustrative implementation, a synchronoustwo-phase-commit process is used, where a transfer between a sender anda receiver is successful before another transfer is attempted. Thisprocess is well known to those in the art, but other processes, forexample asynchronous transfers, can be advantageously employed. Inaddition, as known to those skilled in the art, the data structure shownin FIG. 2 is exemplary for understanding, but the actual structure maybe quite different. For example, there might be a structure of keyobjects identified with an SNS ID and an SNS CTR, but they may not bestored in the same structure with the key objects.

According to an embodiment of the present invention, a securityappliance is assigned an appliance ID number within the cluster.Illustratively, a five bit non-negative number is used. These IDs areshared among the cluster members, and each member maintains a heartbeatwith all the other cluster members. That is, each contacts the others ona regular time basis. If one member does not respond, that member isassumed to be offline. When this occurs, the member with the lowestappliance ID number is automatically elected as the coordinator. Asknown to those skilled in the art, other arrangements may be used toelect the coordinator. As described herein, when, a standalone LKMappliance synchronizes with the coordinator, it will receive the datafor encryption key objects from all the clustered security appliances.

FIG. 3 is a schematic block diagram of an LKM appliance 300, 300′ inaccordance with the present invention. A hardened hardware platform isutilized as the LKM appliance that includes an SEP for management ofkeys among a cluster of security appliances. Specifically, the LKMappliance implements secure storage for cryptainers' encryption keysthat are adapted for use with the security appliances. To that end, theLKM appliance implements a key object database (as well as raw storage)to track and manage the encryption keys. Illustratively, newly receivedkeys can be stored (i) in the key object identified space, which isoverlaid onto a file system of the internal memory 210 of FIG. 2, and(ii) as key object insertion entries on a raw storage partition of thememory 210. Alternately, the key objects may be stored on an externaldatabase server in an encrypted form or on a storage system configuredas external storage.

In more detail, the LKM appliance 300 of FIG. 3 comprises one or moreprocessors, e.g., central processing units (CPU 302), a memory 310, oneor more network adapters 320, SEP 390 and a storage controller 325interconnected by a system bus 340, such as a conventional PeripheralComponent Interconnect (PCI) bus. The SEP 390 is configured to performencryption and decryption operations for the LKM appliance in a securemanner; for example, the SEP 390 may be configured to protect plaintextencryption keys from software executing on each CPU 302. Accordingly,the SEP 390 is illustratively embodied as a FIPS certified module thatis mounted onto a dedicated interface card or other similar card.

The network adapters 320 couple the LKM appliance 300 to one or moresecurity appliances 200 over point-to-point links, wide area networks,virtual private networks implemented over a public network (Internet) orshared local area networks.

The memory 310 illustratively comprises storage locations that areaddressable by the processors and adapters for storing softwareprocesses and data structures associated with the present invention. Theprocessor and adapters may, in turn, comprise processing elements and/orlogic circuitry configured to execute the processes and manipulate thedata structures. In alternative embodiments, the processes may beimplemented in software, hardware, firmware and/or a combinationthereof. As such, the description of processes being implemented insoftware should be taken as exemplary only. It will be apparent to thoseskilled in the art that other processing and memory means, includingvarious computer readable media, may be used for storing and executingprogram instructions pertaining to the invention described herein.

The processes include, e.g., a graphical user interface (GUI) process330, a database (DB) process 335, a file system process 340, a storagedevice control process 345 and a key object synchronization process 350.Illustratively, each of these processes may be implemented as part of anoperating system (not shown) executing on the LKM appliance 300.However, in alternative embodiments, each of these processes may beindependent of an operating system and may operate, e.g., as a user modeapplication on the LKM appliance 300.

The storage device control process 345 may be utilized in managing thestorage controller 325 and storage space 380 provided by internal disks360. Illustratively, the storage device control process 345 may alsomanage external storage, such as that provided by an external storagesystem 110.

The GUI process 330 provides a graphical user interface foradministrators to configure the key management functionality of the LKM300. In alternative embodiments other forms of user interfaces, such asa command line interface (CLI), may be implemented. The database (DB)process 335 implements a database for use in storing and managing keys.Illustratively, the database process 335 implements a database providedon storage space 380 disposed on internal storage devices 160 manage bythe storage controller 325. Storage space 380 may include a partitionadapted to hold a file system implemented by file system process 340 inconjunction with storage device control process 345. In the example ofprocesses executing as part of a conventional BSD (Berkeley SoftwareDistribution) kernel, the file system process 340 may implement aconventional UFS (Unix file System) file system over all or a portion ofstorage space 380 managed by storage controller 325. Alternatively, theDB process 335 may interact with an external database server 130 toimplement storage of keys on the server 130. In embodiments utilizing anexternal storage system 110 (FIG. 1), the DB process 335 may implement adatabase on storage space provided by the external storage system 110.Illustratively, the DB process 335 implements a conventional SQLdatabase; however in alternative embodiments, other data storage and/orretrieval mechanisms may be utilized.

The key object synchronization process 350 illustratively implementsfunctionality to distribute keys between pairs of LKM appliances 300 and300′ and between LKMs and a coordinator security appliance installedwithin a common network environment. Thus, for example, the keysynchronization process ensures that new keys, entered into one LKMappliance 300 within the environment, are distributed (i.e.synchronized) to the other LKM appliance within the environment. Ifthere are more than two LKM appliances, all the pairs are synchronizedindependently. The synchronization process is, illustratively, splitinto a SNS Sync Process 351, and an Object Sync Process 353.

Key synchronization, as discussed below, performs atomic storage toensure that the key objects will not be corrupted. A sequence ofoperations performed atomically is one in which either all the containedoperations fail, or all the contained operations succoed.

The Key Object Synchronization Process 350 includes a local SNS sequencecounter 359, an SNS Peer Map 355 and a table or list 357 of SNS IDs andcorresponding SNS CTRs associated with the peers. The synchronizationprocess is controlled by use of an SNS in the receiving LKM and the SNSID and an SNS CTR 357 for each peer SNS ID, and the local sequencecounter 359 for the receiving LKM, as described below.

Generally, as mentioned above, the process for synchronizing encryptionkey objects ensures that up-to-date keys are stored in the LKMappliances, 300, 300′. To initiate this synchronization, a first LKMappliance (or node) queries a second LKM appliance or the coordinatorsecurity appliance (a peer node) asking it to send its list of SNS IDs.The first LKM then, after receiving an SNS ID, finds (from the Peer Map355) which peer should be queried with respect to the received SNS ID.The first LKM sends the SNS ID to that peer and the highest sequencenumber SNS CTR 357 known to the first LKM appliance that is associatedwith the that SNS ID. The highest sequence number, presumably from anearlier synchronization, corresponds to the key object received fromthat peer most recently. The peer in the meantime might have receivedadditional key objects that are associated with that SNS ID. When thepeer received highest sequence number from the LKM, the peer returns akey object from its key object database that has an equal or higher SNSCTR sequence number. That key object is stored in the first LKMappliance with the received SNS CTR sequence number. This processrepeats between the LKM appliances in both directions until each isfully synchronized with the other LKM appliances—each having up-to-datevalid key objects.

Duplicate keys are deleted from the LKM appliances and thesynchronization will stop (converge) when no new keys are generated.Those skilled in the art would understand that the stored key objectsare not corrupted, so the various storage processes described hereinshould be accomplished via atomic operations, as known to those skilledin the art.

The local sequence counter 359 contains sequence numbers allocated toobjects, preferably in a monotonically increasing sequence, wherein notwo objects within an SNS share the same sequence number. The counter ispreferably implemented in the memory, but may be implemented in otherfunctional locations, as one skilled in the art would understand. In onepreferred embodiment, the sequence counter is 64 bits long, but otherlengths may be used to advantage in other applications. In addition, asdescribed herein, if an existing key object is modified, a versionnumber, associated with the key, is incremented. If a new key is createdand transferred to the LKM appliance, the next higher SNS CTR sequencenumber and a version number are associated with it.

SNS IDs and SNS CTRs 357 reside preferably in persistent memory 310 andthe SNS CTRs contain a count representing the highest sequence countfrom key objects previously synchronized with its peer appliance. Asmentioned before, the persistent memory (data remains intact when notpowered) allows the key synchronization to resume where it left offrather than starting at the beginning again. The various SNS CTRsrepresent the state of synchronization among the cluster and thestandalone nodes. Note that each LKM appliance has is own local sequencecounter and SNS CTRs that co-exist with those of other appliances.

FIG. 4 is a block diagram of the content fields of a key object 400 thatmay be synchronized advantageously as described herein. Each object 400has a sequence number 401 that is a non-negative integer. An Object ID403 is a non-negative integer that is a globally unique identifier ofthis particular object. Illustratively, the integer may be 128 bitslong. An Object data 405 is a field representing the object itself, forexample, if the object is a key, the Object Data 405 might be the keyitself. Additionally, there may be a version number 407 that tracksmodifications to the same key.

The present invention, as mentioned above, involves synchronizing of SNSIDs, and to help enable such synchronizing, as explained below, an SNSPeer Map (shown in FIG. 5), is created at run time and maintained by theSNS Sync Process. When objects are being synchronized among peers, theObject Sync Process, explained below, uses the SNS Peer Map to routequeries to the proper peer. There is one entry in the map per SNS ID501. Each entry contains the peer 503 that last returned a particularSNS ID 505 to the SNS Sync Process.

FIG. 6 is a flow chart illustrating the SNS Sync Process. The processstarts at step 601 and proceeds to step 603 where a node N, typically anLKM, periodically asks a peer, P, for its list of SNS IDs. In step 605,P returns to N its list of SNS IDs. In step 607, for each received SNSID, N determines if the SNS ID is new or already stored at N. If new, instep 609 N adds the new SNS ID to its list and creates an SNS CTR withcontents of zero. If the received SNS ID is old, step 611 is performedwhere N sets the Peer Map to associate the SNS ID with P. The processenters the wait step 613 before returning to the start 601. The waitstep 613 prevents the queries from flooding the system.

FIG. 7 illustrates the Object Sync Process for synchronizing a node withan SNS ID found at a particular peer determined from the Peer Map. Asdescribed herein, this synchronization occurs, illustratively, between anode having an LKM and a peer node having an LKM or a coordinatorsecurity appliance.

Steps in the FIG. 7 flow chart describe a node synchronizing objectsfrom a particular SNS ID within the selected peer. The process starts atstep 701, where a particular node, N1, determines to synchronize itsobjects, typically key objects, with an SNS ID(a). At step 703, N1searches its Peer Map for the peer, P1, associated with that SNS ID(a).In step 705, N1 sends to SNS ID(a) to P1 and the associated SNS CTR (359in FIG. 3) contents, S. In step 707, P1 searches its storage for anobject indexed under SNS ID(a) with a local SNS CTR contents at leastequal to S. If there is no object, in step 711 P1 sends a negativeresponse, NAK, back to N1 in step 713, and the process returns to start701 after waiting in step 715.

In step 709, if P1 has an object with an SNS CTR contents equal to orgreater than S, then, in step 717, P1 sends the object to N1 with theSNS CTR contents, say X, associated with that object. In step 719, N1searches its data store for objects with the same Object ID as thereceived object. If one is found in step 720 and if the received versionnumber is greater than the stored version number in step 721, then N1deletes the stored object and proceeds to step 725. If the receivedversion is not higher that the stored version, the process continues atstep 729 where N1 sets the SNS CTR to X+1.

Otherwise, in step 725, N1 assigns its local sequence number L1 to theobject and, is in step 727, stores the object and increments L1 In step729, N1 sets S to X+1, and returns to start 701.

It should be understood that above-described embodiments are beingpresented herein as examples and that many variations and alternativesthereof are possible. Accordingly, the present invention should beviewed broadly as being defined only as set forth in the hereinafterappended claims. A person of ordinary skill in the art would understandthat the LKM appliance can be implemented as a management console thatmanages operations of the storage system 110. Further, the securityappliance can be implemented as part of the operating system executed atthe storage system 110.

What is claimed is:
 1. A system for synchronizing encryption key objectsbetween an appliance and a peer, the encryption key objects generatedwithin a cluster of security appliances, the system comprising: asequence number space (SNS) within a memory of the appliance having aprocessor, the SNS including a SNS identifier (ID) that identifies apeer, the SNS utilized to store for storing objects created by the peer,wherein the appliance is further configured to create a peer map toroute a query to the peer, wherein the peer map associates the peer withthe SNS; a first sequence counter for the SNS ID maintained at theappliance, the first sequence counter configured to represent a firstnumber of the encryption key objects previously received by theappliance from the peer and stored in the SNS; a second sequence counterassociated with the SNS ID and configured to represent a second numberof the encryption key objects stored in a peer SNS of the peer; and anobject synchronization process, executed on the appliance, andconfigured to transmit to the peer a value of the first sequence countercorresponding to the SNS ID of the appliance, wherein the peer isconfigured to only transmit to the appliance the encryption key objectsstored within the peer SNS of the peer that are associated with a valueof the second sequence counter that is equal to or higher than thetransmitted value of the first sequence counter.
 2. The system of claim1 further comprising a unique identifier associated with the appliance,the peer, and each stored encryption key object.
 3. The system of claim1, wherein the appliance is further configured to store the transmittedencryption key objects at the SNS, the appliance further configured toincrement the value of the first sequence counter in response to storingthe transmitted encryption key objects.
 4. A system for synchronizingencryption key objects between a first node and a second node,comprising: the first node having a processor and a memory, andconfigured to maintain a first sequence number space (SNS) having afirst SNS identifier (ID), the first SNS being associated with thesecond node; the first node maintaining a sequence number counterassociated with the first SNS ID, the sequence number counter indicatinga number of the encryption key objects the first node previouslyreceived from the second node; the first node maintaining at least onepeer map, wherein the peer map associates the second node with the firstSNS ID; the second node maintaining a local sequence counterrepresenting a number of the encryption key objects created by thesecond node; and the first node configured to transmit to the secondnode the sequence number counter, wherein the second node is configuredto only transmit to the first node the encryption key objects stored atthe second node that have local sequence counter values that are equalto or higher than the sequence number counter associated with the firstSNS ID indicating the number of the encryption key objects the firstnode previously received from the second node.
 5. The system of claim 1further comprising a version number associated with each storedencryption key object, the version number incremented in response to amodification of the encryption key object.
 6. The system of claim 1wherein the appliance is further configured to create a new SNS, and anew sequence counter associated with the new SNS in response toreceiving at least one encryption key objects that is not currentlystored in the first SNS.
 7. A method for synchronizing encryption keyobjects, comprising: assigning one or more sequence number spaces withina first node for storing encryption key objects associated with thefirst node and with different nodes: consulting a peer map thatassociates a second node with the one or more sequence number spaces;requesting that a selected different nodes send a list of sequencenumber spaces to the first node; receiving, at the first node, the list;transmitting, to a node identified from the list, a first sequencenumber space identifier and a sequence number space counter valueassociated with the identified node, wherein the sequence number spacecounter value represents a number of encryption key objects that arestored in a first identified sequence number space of the first node andpreviously received from the identified node; receiving, at the firstnode from the identified node, at least one encryption key object storedat the identified node that has a count that is equal or a higher thanthe transmitted sequence number space counter value; and storing the atleast one encryption key object and an updated sequence number spacecounter value in the first identified sequence number space at the firstnode.
 8. The method of claim 7 further comprising associating a uniqueidentifier with each of the first node, the identified node, and each ofthe stored encryption key objects.
 9. The method of claim 7 furthercomprising: electing a coordinator security appliance as the identifiednode, wherein the coordinator security appliance is the only node thatresponds to a request from the first node to synchronize the storedencryption key objects.
 10. The method of claim 9, further comprising:incrementing the sequence number counter with each stored encryption keyobject.